[Refactor][Quantization]KVCache quantization dtype specified by the --kv-quant-dtype parameter by lcfenglinwan · Pull Request #15514 · vllm-project/vllm-ascend

lcfenglinwan · 2026-09-01T14:33:57Z

What this PR does / why we need it?

This PR refactors the flow so the KV cache quantization dtype is explicitly controlled by the user-facing --kv-cache-dtype / --attention-config.indexer_kv_dtype arguments at service startup:

Source of truth is the CLI parameter. The kv cache / SFA C8 / indexer C8 dtypes are now derived from cache_config.cache_dtype / attention_config.indexer_kv_dtype via kv_cache_dtype_str_to_dtype()
(plus checks for "fp8" / "int8"), instead of the hardware-profile-based inference in dsa_attn_kv_plan / DeepseekV4Indexer, which are removed. - Static quantization becomes mandatory. For GQA/MLA-related models,
wrong or missing quantization config now raises a clear ValueError on startup instead of silently falling back to unquantized execution (attention_v1.py, quantization/utils.py::enable_fa_quant).
enable_sparse_*_c8 gating is tied to the actual quantized dtype, so the C8 sparse enter/exit layouts only activate when the dtype is fp8/int8.
Two new patches make this work on vLLM builds that lack the pluggable kv-cache-dtype mechanism (e.g. releases/v0.27.1):
- worker/patch_kv_cache_dtype.py: flips STR_DTYPE_TO_TORCH_DTYPE["fp8"] to torch.float8_e4m3fn in every worker when register_kv_cache_dtype is unavailable (no-op on kvquant_27+).
- platform/patch_indexer_kv_dtype.py: widens the pydantic Literal IndexerKVDType with "int8" so DeepSeek V4's int8 indexer cache is accepted at config validation time.

Does this PR introduce any user-facing change?

Yes — user-facing behavior around quantization becomes explicit:

KV quantization dtype is now chosen by --kv-cache-dtype and the indexer cache dtype by --attention-config.indexer_kv_dtype (values fp8 / int8, previously inferred from the hardware profile).
indexer_kv_dtype now also accepts int8 (Ascend DeepSeek V4 indexer Kcache), which upstream vLLM currently rejects.
GQA/MLA static quantization is now mandatory: if quantization is configured for such models without a matching --kv-cache-dtype fp8|int8 and the corresponding quantized weights, vLLM refuses to start with a clear error instead of silently running unquantized.

How was this patch tested?

vLLM main: vllm-project/vllm@84030bb

github-actions · 2026-09-01T14:35:18Z

👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:‌‌

A PR should do only one thing, smaller PRs enable faster reviews.
Every PR should include unit tests and end-to-end tests ‌to ensure it works and is not broken by other future PRs.
Write the commit message by fulfilling the PR description to help reviewer and future developers understand.

If CI fails, you can run linting and testing checks locally according Contributing and Testing.

gemini-code-assist · 2026-09-01T14:43:13Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refactors the KV cache and indexer dtype handling for Ascend-based models. It introduces robust mechanisms to ensure correct dtype resolution (specifically for 'fp8' and 'int8') across different vLLM versions, including a platform-level patch to support 'int8' indexer caches for models like DeepSeek V4. These changes improve kernel compatibility and ensure consistent configuration validation.

Highlights

KV Cache Dtype Handling: Updated KV cache dtype resolution to use 'fp8' or 'int8' directly from vllm_config, improving compatibility with Ascend kernels.
Indexer KVDType Patching: Added a platform patch to widen the accepted 'indexer_kv_dtype' Literal to include 'int8', enabling support for DeepSeek V4 models on Ascend.
Worker-side Dtype Flip: Implemented a worker-level patch to ensure 'fp8' resolves to 'torch.float8_e4m3fn' in environments lacking the pluggable kv-cache-dtype mechanism.
Codebase Cleanup: Refactored attention and model configuration logic to rely on standard vllm_config properties, removing redundant or hardcoded logic.

New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

Suggested PR Title:

[Attention][Feature] Support int8 indexer KV cache dtype and pluggable fp8 kv-cache-dtype fallback

Suggested PR Summary:

### What this PR does / why we need it?
This PR introduces support for `int8` indexer KV cache dtypes, which are required by DeepSeek V4 and similar Ascend sparse-attention models. It implements a platform patch to dynamically widen the Pydantic validation schema for `AttentionConfig.indexer_kv_dtype` to accept `"int8"`. It also adds a worker patch to map `"fp8"` to `torch.float8_e4m3fn` on older vLLM releases lacking the pluggable `register_kv_cache_dtype` mechanism.

However, several critical issues were identified in the review:
- **Crash in non-quantized models**: `enable_fa_quant` raises an empty `ValueError("")` instead of returning `False`, which will crash model initialization for non-quantized models.
- **AttributeErrors**: `self.vllm_config` is accessed in `AscendConfig` and `DeepseekV4IndexerCache` where it is not set or should be accessed via the passed parameter.
- **Uninitialized attributes**: In `sfa_v1.py` and `model_runner_v1.py`, cache dtype attributes are left uninitialized if `indexer_kv_dtype` is `"bf16"` while `enable_sparse_sfa_c8` is `True`.
- **Unconditional call**: `kv_cache_dtype_str_to_dtype` is called unconditionally in `mla_v1.py` which can cause runtime errors for non-quantized models.

### Does this PR introduce _any_ user-facing change?
Yes, users can now specify `--attention_config.indexer_kv_dtype int8` for DeepSeek V4 models on Ascend.

### How was this patch tested?
No tests were provided in the diff. It is recommended to add integration tests for DeepSeek V4 with `int8` indexer KV cache.

github-actions · 2026-09-02T01:52:36Z